Coordinate computation device and method, and an image processing device and method

ABSTRACT

In an image processing for correcting a distorted image obtained by photography by use of a super-wide angle optical system such as a fisheye lens or an omnidirectional mirror, to obtain an image of a perspective projection method, of a composite index (Rn) combining a height on the projection sphere with a distance from the optical axis is computed ( 301 ), and a distance (Rf) from an origin in the distorted image is computed ( 302 ), using the composite index (Rn). Further, two-dimensional coordinates (p, q) in the distorted image are computed ( 303 ), using the distance (Rf) from the origin, and a pixel value in an output image is determined using a pixel at a position in the distorted image specified by the two-dimensional coordinates, or a pixel or pixels neighboring the specified position. It is possible to perform the projection from the projection sphere to the image plane, that is, the computation of the coordinates on the coordinate plane, while restricting the amount of computation.

TECHNICAL FIELD

The present invention relates to an image processing device and methodfor correcting a distorted image, obtained by photography with asuper-wide angle optical system such as a fisheye lens or anomnidirectional mirror to obtain an image of a perspective projectionmethod, and to a coordinate computation device and method used in suchimage processing device and method.

BACKGROUND ART

Super-wide angle optical systems such as fisheye lenses andomnidirectional mirrors can project an image with an angle of view of180 degrees or more, onto a single image plane, and is used in variousareas in which a wide photographic field of view is required.

In an image obtained toy photography using a super-wide angle opticalsystem, the shapes of the captured objects appear distorted by barreldistortion, in comparison with an image of a normal perspectiveprojection, so that the captured image (distorted image) gives an oddimpression if it is displayed without alteration. Therefore, when avideo obtained using a super-wide angle optical system is displayed, aregion of interest, which is a part of the image, is extracted, and theextracted image is transformed to an image of a normal perspectiveprojection method, before the display.

In this transformation, a projection surface (a projection sphere) on ahemispherical surface on one side of the image plane is assumed, and aprojection is made from the projection sphere to the image plane. Thisprojection includes a process of determining the position on the imageplane corresponding to a point of interest on the projection sphere.Patent Reference 1 shows use of the zenith angle for the computation ofthe image height (distance from the optical axis) on the image plane, inthis process (column 7). If the zenith angle is used, it is necessary todetermine a trigonometric function of the zenith angle, when, forexample, the projection is performed by orthographic projection,stereographic projection, or equisolid angle projection. PatentReference 1 discloses the use of a lookup table for simplifying theprocess of determining the trigonometric function.

PRIOR ART REFERENCES Patent References

-   Patent Reference 1: Japanese Patent No. 3126955

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

Use of the position in the direction of the optical axis (height)instead of the zenith angle may also be considered. When the height isused, the computation of the trigonometric function becomes unnecessaryand the amount of computation can be reduced. However, in the vicinityof the zenith of the projection sphere, the variation in the height fora given variation in the zenith angle becomes small, and it is necessaryto increase the number of bits of the data representing the height inthe vicinity of the zenith, in order to perform the computation with thedesired precision.

Use of the distance from the optical axis can also be considered. Inthis case as well, the computation of the trigonometric function becomesunnecessary, and the amount of computation can be reduced. However, inthe vicinity of the rim of the projection sphere, the variation in thedistance from the optical axis for a given variation in the zenith anglebecomes small, and if is necessary to increase the number of bits of thedata representing the distance in the direction of the optical axis lathe vicinity of the rim, in order to perform the computation with thedesired precision.

The present invention addresses the problems described above, and itsobject is to enable projection from the projection sphere to the imageplane, that is, computation of the coordinates, on the coordinate plane,with high precision, while restricting the amount of the computation.

MEANS FOR SOLVING THE PROBLEM

A coordinate computation device according to the present inventiontransforms three-dimensional coordinates on a projection sphere totwo-dimensional coordinates on a distorted image, and comprises:

a composite index computation unit tor computing a composite index whichis a combination of a height of a point of interest on the project ionsphere and a distance of the point of interest from an optical axiswhich are obtained from said three-dimensional coordinate;

a distance computation unit for computing a distance from an origin insaid distorted image, of a point in said distorted image correspondingto said point of interest, from said composite index; and

a coordinate computation unit for computing the two-dimensionalcoordinates in said distorted image, from the distance from said originin said distorted image.

EFFECT OF THE INVENTION

According to the present invention, the composite index, which is acombination of the height on the projection sphere and the distance fromthe optical axis, is used, so that the computation of the coordinatescan be made with high precision, with a small number of bits.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an example of the configuration of animago processing device according to the present invention.

FIG. 2 is a diagram showing an example of a fisheye image.

FIG. 3 is a diagram showing a projection from an object to a projectionsphere, and a projection from the projection sphere to an image plane.

FIG. 4 is a diagram showing x, y, z coordinate values on the projectionsphere.

FIG. 5 is a diagram showing positional relationships among the imageplane, the projection sphere, and an output image plane.

FIGS. 6( a) to (d) are diagrams showing a room magnification factor, apan angle, a tilt angle, and a plane inclination angle.

FIG. 7 is a block diagram showing an example of the configuration of animage correction processing unit in the image processing device in FIG.1.

FIGS. 8( a) and (b) are diagrams showing a summary of an imagecorrection process.

FIGS. 9( a) to (c) are diagrams showing examples of a zenith angle, aheight, and a distance from the optical axis of a point of interest onthe projection sphere.

FIG. 10 is a block diagram showing a coordinate computation device in afirst embodiment of the present invention.

FIG. 11 is a graph showing a relationship between the zenith angle, anda composite index Rn, the height z, and the distance r from the opticalaxis, of a point of interest on the projection sphere.

FIG. 12 is a flowchart showing the processing procedure in a coordinatecomputation method in a second embodiment of the present invention.

MODE FOR CARRYING OUT THE INVENTION First Embodiment

FIG. 1 shows an image processing device in a first embodiment of thepresent invention. The illustrated image processing device has a fisheyelens 101 as an example of a super-wide angle optical system, an imagingsignal generation unit 102, a video memory 103, an image correctionprocessing unit 104, and a corrected video output circuit 105.

The fisheye lens 101 optically captures an image with a wide angle ofview, and, for example, a circular distorted image like the one shown inFIG. 2 is obtained. The circular image in FIG. 2 is an image of theobjects within a range of the angle of view of up to 180 degrees. FIG. 2shows, in a simplified manner, an image obtained when photography ismade from an intersection of streets, in the direction of one of thestreets. In the following description, a distorted image captured by useof the fisheye lens 101 is referred to as a fisheye image.

Imaging with a fisheye lens will be described below with reference toFIG. 3.

It will be assumed that there are an image plane F perpendicular to theoptical axis AL of the fisheye lens 101, and a projection surface(projection sphere) S on a hemisphere centered at the point ofintersection O of the image plane F and the optical axis AL, andpositioned on the object side (photographed space side) of the imageplane F, with its bottom surface coinciding with the image plane.

It can be considered that all light from the object propagates towardthe above-mentioned center O, and is projected onto the projectionsphere S, and is then projected from the projection sphere S onto theimage plane F.

The projection from the projection sphere S to the image plane F iscarried out by one of the orthographic projection method, thestereographic projection method, the equidistance projection method, theequisolid angle projection method and so on. The method by which theprojection is carried out differs depending on the fisheye lens 101 thatis used.

FIG. 3 shows a case in which light from a direction indicated by a chainline BL is projected onto a point Ps on the projection sphere S, and isthen projected onto a point Pf on the image plane F.

The position of the point Pf on the image plane F is represented bytwo-dimensional coordinates (p, q). These two-dimensional coordinatesrepresent the positions in the direct ions of the two coordinate axes,that is, a P axis and a Q axis of the image plane F, that extend in twodirections which are perpendicular to each other.

The position of the point Ps on the projection sphere S is representedby three-dimensional coordinates (x, y, z). These three-dimensionalcoordinates have their origin at the point of intersection O of theimage plane F and the optical axis AL (accordingly, the center of thebottom surface of the above-mentioned hemisphere), and representpositions in the directions of a Z axis, which is a coordinate axisextending in the direction of the optical axis AL, and an X axis and a Yaxis which are coordinate axes respectively coinciding with the P axisand the Q axis on the image plane P. In FIG. 4, the magnitudes of the x,y, z coordinate values are shown separately.

The position of the point Ps can also be represented by a zenith angle θand an azimuth angle ∂.

The zenith angle θ of the point Ps is an angle of inclination from theoptical axis AL, or in other words, the angle formed by the straightline connecting the point Ps and the origin O, and the optical axis AL.

The azimuth angle ∂ of the point Ps is an angle in the direction ofrotation about the optical axis, and is referenced to the direction ofthe X axis, for example.

The projection from the projection sphere S to the image plane F iscarried out in such a way that the azimuth aegis does not change. Theazimuth angle of the point Pf is therefore identical, to the azimuthangle of the point Ps, and is denoted by the reference character ∂.

The relationship between the distance Pf from the optical axis AL to thepoint Pf (accordingly, the distance from the origin O to the point Pf),and the distance r from the optical axis AL to the point Ps differsdepending on the projection method.

That is, the relationship between the zenith angle θ of the point Ps andthe distance Rf from the optical axis AL to the point Pf differsdepending on the projection method as follows. For example, in the caseof the orthographic projection method, there is a relationship:

Rf=Rb×sin θ  (1)

while in the case of the stereographic projection, there is arelationship:

Rf=Rb×tan(θ/2)  (2)

In the above-mentioned formulas, Rb is the radius of the projectionsphere S. The radios Rb of the circular fisheye image shown in FIG. 2(the distance from the center of the image to a point with an angle ofview of 180 degrees) is equivalent to the radius of the projectionsphere S.

Also, the relation between the zenith angle θ and the distance r fromthe optical axis AL to the point Ps is:

r=Rb×sinθ  (3)

Accordingly, in the case of the orthographic projection method, there isa relation:

Rf=r  (4)

while in the case of the stereographic projection method, there is arelation:

$\begin{matrix}\lbrack {{Mathematical}\mspace{14mu} {Expression}\mspace{14mu} 1} \rbrack & \; \\{{Rf} = {{2{{Rb} \cdot {\tan ( \frac{\theta}{2} )}}} = {2{{Rb} \cdot {\tan( \frac{\sin^{- 1}\frac{r}{Rb}}{2} )}}}}} & (5)\end{matrix}$

The imaging signal generation unit 102 includes an imaging elementformed, for example of a CCD, having the image plane F as shown in FIG.3, and generates an electrical image signal representing an opticalimage.

The video memory 103 stores the image signal generated toy the imagingsignal generation unit 102 in units of frames.

The image correction processing unit 104 reads the image signal storedin the video memory 103, corrects the fisheye image represented by theimage signal, and outputs the corrected image. In the correction, apartial region of the fisheye image, that is, a region including aposition in the fisheye image, corresponding to a direction of view DOVshown in FIG. 5, and the positions surrounding it (the part of thefisheye image generated by light from the direction of view DOV and itssurrounding directions) is selected, and the selected image istransformed to an image of a perspective projection method. The imageresulting from the correction by the image correction processing unit104 will be referred to below as an ‘output image’.

The output image is an image obtained by a perspective projection of theimage on the projection sphere S to a plane (output image plane) H, asshown in FIG. 5, and is denoted by the same reference character H as theoutput image plane.

The plane H is tangent to the projection sphere S at the point ofintersection G of the straight line indicating the direction of view DOVand the projection sphere s. The position of a point Ph on the plane(output image plane) H is represented by coordinates (u, v) having theirorigin at the point of tangency G with the projection sphere S. Thecoordinates (u, v) represent the positions in the directions the U axisand the V axis which are mutually perpendicular coordinate axes on theoutput image plane H.

The U axis forms an angle φ with the rotation reference axis J. Therotation reference axis J passes through the origin G, is parallel tothe XY plane, and perpendicular to the straight line OG (the straightline connecting the origin O and the origin G). The angle φ will bereferred to as a plane inclination angle or a rotation angle.

The azimuth angle ∂ of the origin G will be referred to as a pan angle,and is denoted by a reference character α. The zenith angle θ of theorigin G will be referred to as a tilt angle, and denoted by a referencecharacter β. The size of the output image, that is, the angle of view ofthe output image (electronic zoom angle of view) can be varied by amagnification factor (zoom magnification factor) m. The change in thesize of the output image according to the change in the zoommagnification factor is conceptually illustrated in FIG. 6( a). FIGS. 6(b), (c), and (d) separately show the pan angle α, the tilt angle β, andthe plane inclination angle φ for the same output image plane H as thatshown in FIG. 5.

In the projection of the image from the projection sphere S to theoutput image plane H, the point Ps(x, y, z) on the projection, sphere Sis projected to the point Ph(u, v) at which the straight line passingthrough the point Ps and the origin O intersects the output image planeH.

To generate the above mentioned output image (the image projected to theoutput image plane H), the image correction processing unit 104 computesthe position in the fisheye image corresponding to the position of eachpixel (pixel of interest) in the output image, computes the pixel valueof the above-mentioned pixel of interest in the output imago based onthe pixel value of the pixel at the computed position in the fisheyeimage, or the pixel value or values of the pixel or pixels in theneighborhood of the above-mentioned pixel of interest, and generates theimage by an array of pixels having the computed pixel values.

The corrected video output circuit 105 outputs the signal of thecorrected image in the form of a normal television video signal, such asa video signal of an NTSC format. Incidentally, this signal may beencoded by an encoding device, and transmitted to a remote location, atwhich a video display may be made.

The image correction processing unit 104 has, for example, an outputrange selection unit 201, a spherical coordinate computation unit 202, afisheye image coordinate computation unit 203, and a pixel valuecomputation unit 204, as shown in FIG. 7.

The image correction process carried cut by the image correctionprocessing unit 104 will be described with reference to FIG. 5, andFIGS. 8( a) and (b).

The output range selection unit 201 selects the range of the outputimage by selecting the desired direction of view DOV, that is, thedesired pan angle α and the desired tilt angle β, and also selecting thedesired plane inclination angle φ, and the desired size of the outputimage, that is, the desired zoom magnification factor m. The selectionis made according to, for example, user operations. Alternatively, therange of the output image may be automatically varied over time.

The selection of the output range by the output range selection unit 201(FIG. 8( a)) is equivalent to determining or varying the direction ofview and determining or varying the rotation angle by mechanical pan,tilt, rotation (change in the plane inclination angle) of the camera,determining or varying the rotation angle, and determining or varyingthe magnification factor by an optical zoom, and the process performedby the output range selection unit 201 may also foe referred to as anelectronic PTZ (pan, tilt, and zoom) process.

The spherical coordinate computation unit 202 projects the output imageH having its range selected by the output range selection unit 201, ontothe projection sphere S, as shown in FIG. 5, and computes thethree-dimensional coordinates of the point on the projection sphere Scorresponding to each point in the output image H. The computation ofthe three-dimensional coordinates is a process of determining theposition (x, y, z) on the projection sphere S corresponding to, forexample, the position of each point, for example, (u, v), in the outputimage H. The position on the projection sphere S corresponding to eachpoint Ph in the output image is the position of the point ofintersection of a straight line connecting the above-mentioned eachpoint Ph and the origin of and the projection sphere S. The range on theprojection sphere S corresponding to the output image H is denoted byreference characters Sh in FIG. 5 and FIG. 8( a).

The fisheye image coordinate computation unit 203 projects the image(the image on the projection sphere) in the range Sh shown in FIG. 8( a)onto the fisheye image as shown in FIG. 8( b), thereby to compute thefisheye image coordinates. This projection is a process of determiningthe position in the fisheye image, that is, the position (p, q) on theimage plane F, corresponding to the position (x, y, z) of each point Pson the projection sphere S. The range (extracted range) in the fisheyeimage (denoted by the same reference character W as the image plane)corresponding to the range Sh is denoted by reference characters Fh inFIG. 8(b).

The method by which rise projection from the projection sphere S to thefisheye image can be made by any of various methods such as those of theorthographic projection, the equidistance projection, the stereographicprojection, the equisolid angle projection, and so on, and is determineddepending on the projection method of the fisheye lens.

The pixel value confutation unit 204 computes the pixel value of eachpixel in the output image in FIG. 5. In the computation of the pixelvalue, it uses the pixel value of the pixel at the position (p, q) inthe fisheye image corresponding to the position (u, v) of each pixel(pixel of interest) in the output image in FIG. 5, or the pixel value(s)of the pixel(s) in the neighborhood of the position (p, q) (neighboringpixel(s)). That is, if there is a pixel at the position in the fisheyeimage corresponding to the position of the above-mentioned pixel ofinterest, the pixel value of that pixel is used, as the pixel value ofthe above-mentioned pixel of interest, and if there is no pixel at theposition in the fisheye image corresponding to the pixel of interest,the pixel value of the above-mentioned pixel of interest is determinedby interpolation from the pixel value or values of one or more pixels inthe neighborhood of the above-mentioned corresponding position in thefisheye image.

When the projection from the projection sphere S to the image plane F iscarried out in the projection fisheye image coordinate computation unit203, a composite index obtained by combining the height with thedistance from the optical axis is used, according to the presentinvention. The reason for using the above-mentioned composite index willbe explained, after explaining the problems regarding this point.

In general, for determining the image height in the fisheye image F inthis projection, use of the zenith angle, use of a value (height)representing the position in the direction of the optical axis, and useof the distance from the optical axis may be considered.

For example, the zenith angle is used in Patent Reference 1 (FIG. 1, andColumn 7 in the above-mentioned patent reference) as shown in FIG. 9(a). Data indicating the zenith angle θ can be expressed with the sameprecision by the data of the same bit width both in the vicinity of thezenith of the projection sphere, and in the vicinity of the rim of theprojection sphere (image plane). If the zenith angle is used, however,it is necessary to compute a trigonometric function when the method forprojection from the projection sphere to the fisheye image is theorthographic projection method, the stereographic projection method, orthe equisolid angle projection method. In Patent Reference 1, a lookuptable is used to simplify the computation of the trigonometric function.

It may also be considered to use, in place of the zenith angle, theposition (height) to the direction of the optical axis, that is, thecoordinate value x, shown in FIG. 9( b). When the height is used, thecomputation of the trigonometric function becomes unnecessary, and theamount of computation can be reduced. In the vicinity of the zenith ofthe projection sphere, however, the variation in the height for a givenvariation in the zenith angle becomes small, and many bits of precisionare needed only in the vicinity of the zenith. (Data of a larger binwidth needs to be used to perform the computation with the sameprecision.) For example, when the zenith angle varies from 89 degrees to90 degrees in the vicinity of the rim, the variation in the height isapproximately 1.745×10⁻², while when the zenith angle varies from 0degrees to 1 degree in the vicinity of the zenith, the variation in theheight is approximately 1.523×10⁻⁴, so chat the precision differs by afactor of about 100.

It may also be considered to use the distance from the optical axis, asshown in FIG. 9( c). In this case as well, the computation of thetrigonometric function becomes unnecessary and the amount of computationcan be reduced. In the vicinity of the rim of the projection sphere,however, the variation in the distance from the optical axis for a givenvariation in the zenith angle becomes small, so that many bits ofprecision are needed only in the vicinity of the rise. (Data of a largerbit width needs to be used to perform the computation with the sameprecision.) For example, when the zenith angle varies from 0 degrees to1 degree in the vicinity of the zenith the variation in the distancefrom the optical axis is approximately 1.745×10⁻², while when the zenithangle varies from 89 degrees to 90 degrees in the vicinity of the rim,the variation in the distance from the optical axis is approximately1.523×10⁻⁴, so that the precision differs by a factor of about 100.

The coordinate computation device of the present invention has been madein an attempt to solve the problems described above, and has a featurethat the precision of the computation does not vary greatly, even if thebit width of the data is not changed in the vicinity of the zenith orthe rim.

FIG. 10 shows an example of the configuration of a coordinatecomputation device 203 a. The illustrated coordinate computation device203 a is used as the fisheye image coordinate computation unit 203 inFIG. 7, and computes the coordinates in a two-dimensional image.

The coordinate computation device 203 a in FIG. 10 has a composite indexcomputation unit 301, a distance computation unit 302, and a coordinatecomputation unit 303.

The three-dimensional coordinates (x, y, z) representing the position onthe projection sphere corresponding to the position of the pixel ofinterest in the output image are input to the coordinate computationdevice 203 a. The three-dimensional coordinates (x, y, z) are obtainedby selection, by means of the output range selection unit 201, of therange of extraction according to the pan angle α, the tilt angle β andthe plane inclination angle φ, and the room magnification factor, andprojection by means of the spherical coordinate computation unit 202,onto the projection sphere.

The three-dimensional coordinates (x, y, z) are input to the compositeindex computation unit 301.

The composite index computation unit 301 computes the composite index Rnwhich is a combination of the distance r from the optical axis:

r=√{square root over ( )}(x ² +y ²)  (6)

and the height z, according to the following formulas (7a), (7b), (8a)and (8b):

if

z≧√{square root over ( )}(x ² +y ²)  (7a)

then

Rn=√{square root over ( )}(x ² +y ²)  (7b)

and if

z<√{square root over ( )}=(x ² +y ²)  (8a)

then

Rn=√{square root over ( )}2×Rb−z  (8b)

z is equal to or greater than √{square root over ( )}(x²+y²) in therange in which the zenith angle θ is equal to or less than 45 degrees,so that the formulas (7a) and (8a) can be rewritten as the followingformulas (7c) and (8c):

z≧Rb/√{square root over ( )}2  (7c)

z<Rb/√{square root over ( )}2  (8c)

FIG. 11 shows the relationships among the point of interest Ps on theprojection sphere S, the xenith angle (the angle formed by the lineconnecting the point of interest Ps and the original O, and the opticalaxis AL), and the composite index Rn, the height z and the distance rfrom the optical axis.

In FIG. 11, the horizontal axis represents the zenith angle θ of thepoint of interest Ps.

The composite index Rn, the height z, and the distance r from theoptical axis are shown on the vertical axis.

As shown in FIG. 11, the slops of the height z becomes zero when thezenith angle is 0 degrees, so that the variation in the value of theheight with respect to the zenith angle is small in the vicinity of 0degrees, and a large bit width becomes necessary to carry out thecomputation using the height with high precision.

Also, the slope of the distance r from the optical axis becomes herowhen the zenith angle θ is 30 degrees, so that the variation in thevalue of the distance r from the opt teal axis with respect to thezenith angle θ is small in the vicinity of 90 degrees, and a large bitwidth becomes necessary to carry out the computation with high precisionby using the distance r from the optical axis.

In contrast, the slope of the composite index Rn does not vary so much,throughout the whole range of the zenith angle, so that the bit widthneed not be large.

Also, at the point where the height z and the distance r from theoptical axis are equal (the boundary between the range within which theformula (7a) is satisfied and the range within which the formula (8a) issatisfied), Rn given by the formula (7b) and Rn given by the formula(8b) are equal, and the slope of Rn given by the formula (7b) and theslope of Rn given by the formula (8b), that is, the slope of −z, and theslope of r (=√{square root over ( )}(x²+y²) are equal, so that thecomposite index Rn obtained by combining the two different parameters (zand r) varies continuously and smoothly.

The composite index Rn is sent to the distance computation unit 302.

The distance computation unit 302 determines the distance (image height)Rf of the point Pf from the origin in the fisheye image based on thecomposite index Rn, by computation represented by the followingformulas. That is, if the image height Rf is represented by:

Rf=Rb·R(θ)  (9)

as a function of the radius Rb of the projection sphere and the zenithangle θ, then in the range in which

z≧√{square root over ( )}(x ² +y ²)  (7a)

is satisfied, the distance computation unit 302 determines Rf by:

$\begin{matrix}\lbrack {{Mathematical}\mspace{14mu} {Expression}\mspace{14mu} 2} \rbrack & \; \\{{Rf} = {{Rb} \cdot {F( {\cos^{- 1}\frac{\sqrt{{Rb}^{2} - {Rn}^{2}}}{Rb}} )}}} & (10)\end{matrix}$

whereas in the range in which

z<√{square root over ( )}(x ² +y ²)  (8a)

is satisfied, the distance computation unit 302 determines Rf by:

$\begin{matrix}\lbrack {{Mathematical}\mspace{14mu} {Expression}\mspace{14mu} 3} \rbrack & \; \\{{Rf} = {{Rb} \cdot {F( {\cos^{- 1}\frac{{\sqrt{2} \times {Rb}} - {Rn}}{Rb}} )}}} & (11)\end{matrix}$

The reason, why Rf is determined by the formula (10) and the formula(11) will be described below.

When the point Ps on the sphere is projected to the point Pf in thefisheye image, the image height Rf of the point Pf is represented as afunction of Rb and the zenith angle θ;

Rf=Rb·F(θ)  (12)

In the case of the orthographic projection, for example, it isrepresented by:

Rf=Rb·sin(θ)  (12a)

whereas in the case of the stereographic projection, it is representedby:

Rf=2Rb·tan(θ)  (12b)

θ in the formula (12) is represented by:

θ=cos⁻¹(z/Rb)  (13)

From the formula (12) and the formula (13),

$\begin{matrix}\lbrack {{Mathematical}\mspace{14mu} {Expression}\mspace{14mu} 4} \rbrack & \; \\{{Rf} = {{Rb} \cdot {F( {\cos^{- 1}\frac{z}{Rb}} )}}} & (14)\end{matrix}$

is obtained.

Rb is related to x, y and z by:

Rb ² =x ² +y ² +z ²  (15)

Accordingly,

z ² =Rb ²−(x ² +y ²)  (16)

Also, the formula (7b) holds in the range in which the formula (7a) issatisfied, so that

Rn ²=(x ² +y ²)  (17)

From the formula (17) and the formula (16),

z ² =Rb ² −Rn ²  (18)

Accordingly,

z=√{square root over ( )}(Rb ² −Rn ²)  (19)

From the formula (14) and the formula (19),

$\begin{matrix}\lbrack {{Mathematical}\mspace{14mu} {Expression}\mspace{14mu} 5} \rbrack & \; \\{{Rf} = {{Rb} \cdot {F( {\cos^{- 1}\frac{\sqrt{{Rb}^{2} - {Rn}^{2}}}{Rb}} )}}} & (10)\end{matrix}$

is obtained.

In the range in which the formula (8a) as satisfied, the formula (8b)holds, so that,

z=√{square root over ( )}2×Rb−Rn  (20)

From the formula (14) and the formula (20),

$\begin{matrix}\lbrack {{Mathematical}\mspace{14mu} {Expression}\mspace{14mu} 6} \rbrack & \; \\{{Rf} = {{Rb} \cdot {F( {\cos^{- 1}\frac{{\sqrt{2} \times {Rb}} - {Rn}}{Rb}} )}}} & (11)\end{matrix}$

is obtained,

From the above, it can be seen that Rf is given by the formula (10) whenthe formula (7a) is satisfied, and by the formula (11) when the formula(8a) is satisfied.

In the case of the orthographic projection, that is, when Rf is given bythe formula (12a),

Rf=Rb·sin(cos⁻¹(z/Rb))  (14a)

from the formula (12a) and the formula (13).

From the formula (14a) and the formula (19).

$\begin{matrix}\lbrack {{Mathematical}\mspace{14mu} {Expression}\mspace{14mu} 7} \rbrack & \; \\{{Rf} = {{Rb} \cdot {\sin( {\cos^{- 1}\frac{\sqrt{{Rb}^{2} - {Rn}^{2}}}{Rb}} )}}} & ( {10a} )\end{matrix}$

From the formula (14a) and the formula (20),

$\begin{matrix}\lbrack {{Mathematical}\mspace{14mu} {Expression}\mspace{14mu} 8} \rbrack & \; \\{{Rf} = {{Rb} \cdot {\sin( {\cos^{- 1}\frac{{\sqrt{2} \times {Rb}} - {Rn}}{Rb}} )}}} & ( {11a} )\end{matrix}$

In the case of the stereographic projection, that is, when Rf is givenby the formula (12b),

Rf=2Rb·tan(cos⁻¹(z/Rb))  (14b)

from the formula (12b) and the formula (13).

From the formula (14b) and the formula (19),

$\begin{matrix}\lbrack {{Mathematical}\mspace{14mu} {Expression}\mspace{14mu} 9} \rbrack & \; \\{{Rf} = {2{{Rb} \cdot {\tan( {\cos^{- 1}\frac{\sqrt{{Rb}^{2} - {Rn}^{2}}}{Rb}} )}}}} & ( {10b} )\end{matrix}$

From the formula (14b) and the formula (20),

$\begin{matrix}\lbrack {{Mathematical}\mspace{14mu} {Expression}\mspace{14mu} 10} \rbrack & \; \\{{Rf} = {2{{Rb} \cdot {\tan( {\cos^{- 1}\frac{{\sqrt{2} \times {Rb}} - {Rn}}{Rb}} )}}}} & ( {11b} )\end{matrix}$

The coordinate computation unit 303 computes the two-dimensionalcoordinates (p, q) on the fisheye image, from the image height Rfcomputed by the formula (10) and the formula (11), and thethree-dimensional coordinates (x, y, z) on the projection sphere. Theformulas for the computation are as follows:

If

Rf≠0  (21Aa),

then

p=x×Rf/r  (21Ab)

q=y×Rf/r  (21Ac)

If

Rf=0  (21Ba),

then

p=0  (21Bb)

q=0  (21Bc)

The case in which the image height Rf is used by the computationrepresented by the formula (10) and the formula (11), in the distancecomputation unit 302 has been described above, but Rf may also bedetermined by using a lookup table storing the relationship between Rnand Rf.

In this case, if values of Rf ware to be stored for all the values ofRn, a lookup table of a large capacity would be necessary. To avoidthis, a lookup table storing values DF[0] to DF[N−1] of Rf correspondingto a plurality of, namely, N discrete representative values DN[0] toDN[N−1] of Rn may be provided, and Rf corresponding to Rn may bedetermined by interpolation computation using the value of Rn, thenearest representative value DN[n] which is not greater than Rn, thenearest representative value DN[n+1] which is greater than Rn, andoutput values DF[n], DF[n+1] read out when DN[n], DN[n+1] are input. Thecomputational formula used in that case is shown as the formula (22).

$\begin{matrix}\lbrack {{Mathematical}\mspace{14mu} {Expression}\mspace{14mu} 11} \rbrack & \; \\{{Rf} = {{{DF}\lbrack n\rbrack} + {\frac{( {{{DF}\lbrack {n + 1} \rbrack} - {{DF}\lbrack n\rbrack}} )}{( {{{Dn}\lbrack {n + 1} \rbrack} - {{DN}\lbrack n\rbrack}} )} \times ( {{Rn} - {{DN}\lbrack n\rbrack}} )}}} & (22)\end{matrix}$

Here, the input values DN[0] to DN[N−1] are such that they monotonicallyincrease with respect to increase in n (n=0 to N−1) That is, DN[0] toDN[N−1] are mutually interrelated by the formula below.

DN[n]≦DN[n+1]  (23)

(where, n is any value from 0 to N−2).

n in the formula (22) satisfies the condition represented by thefollowing formula (24).

DN[n]≦Rn<DN[n+1]  (24)

Incidentally, when Rn is equal to or greater than DN[N−1] then DF[N−1]is output as Rf.

In general, the N values of the above-mentioned DN[n] and DF[n] are setwithin the range of values that can be taken by Rn and Rf in such a waythat the relationship between Rn and Rf can be approximated by apolyline, using the formula (10) in the range in which the condition ofthe formula (7a) is satisfied and using the formula (11) in the range inwhich the condition of the formula (8a) is satisfied, and in such a waythat, for example, the differences between DN[n] and DN[n+1] areconstant.

When the orthographic projection is used, the formula (10a) and theformula (11a) are used as the formula (10) and the formula (11), whereaswhen the stereographic projection is used, the formula (10b) and theformula (10b) are used as the formula (10) and the formula (11).

Incidentally, when the relationship between Rn and Rf cannot beexpressed by a computational formula, the relationship between Rn and Rfis determined by measurement. For example, the relationship between Rnand Rf can be determined by photographing a two-dimensional square gridchart, or the like by use of the fisheye lens, and investigating theamount of distortion of the square grid on the photographed image.

The present invention has been described above as a coordinatecomputation device and an image processing device equipped with thecoordinate computation device, but a coordinate computation method andan image processing method implemented by these devices also form partof the present invention.

Second Embodiment

FIG. 12 shows a processing procedure in the coordinate computationmethod implemented by the coordinate computation device in FIG. 10.

In a step ST1 in FIG. 12, a composite index Rn which is a combination ofthe height and the distance from the optical axis is computed from thethree-dimensional coordinates (x, y, z) on the projection sphere,according to the formula (7a), the formula (7b), the formula (8a), andthe formula (8b) shown in the first embodiment.

Next, in a step ST2, the distance Rf from the origin in the fisheyeimage is computed from the composite index Rn.

This process may implemented by performing the computation representedby the formula (10) and the formula (11), or may be implemented by useof a lookup table.

Next, in a step ST3, the two-dimensional coordinates (p, q) on thefisheye image are computed from the distance Rf from the origin in thefisheye image, and the three-dimensional coordinates (x, y, z) on theprojection sphere, according to the formulas (21Aa) to (21Bc) shown inthe first embodiment.

By transforming the three-dimensional coordinates on the projectionsphere to the two-dimensional coordinates on the fisheye image, usingthe above-described method, the transformation can be made with highprecision and with a small bit width.

The coordinate computation method, shown in FIG. 12 may be implementedby software, that is, by a programmed computer.

The present invention has been described above for the case in whichpart of a fisheye image is selected and transformed to an image of aperspective projection method, but the invention may be applied toprocesses other than that described above, for example, the presentinvention may be applied to a process for obtaining a panoramic imagefrom a fisheye image.

The case in which a circular distorted image is obtained by use of afisheye lens has been described above, but the present invention mayalso be applied where an annular distorted image is obtained.

The coordinate computation device according to the present invention orthe image processing device equipped with the coordinate computationdevice may be utilized in a surveillance system.

REFERENCE CHARACTERS

201 output range selection unit; 202 spherical coordinate -computationunit; 203 fisheye image coordinate computation unit; 203 a coordinatecomputation device; 204 pixel value computation unit; 301 Compositeindex computation unit 302 distance computation unit; 303 coordinatecomputation unit.

1. A coordinate computation device that transforms three-dimensionalcoordinates on a projection sphere to two-dimensional coordinates on adistorted image, comprising: a composite index computation unit forcomputing a composite index which is a combination of a height of apoint of interest on the projection sphere and a distance of the pointof interest from an optical axis which are obtained from saidthree-dimensional coordinates; a distance computation unit for computinga distance from an origin in said distorted image, of a point in saiddistorted image corresponding to said point of interest, from saidcomposite index; and a coordinate computation unit for computing thetwo-dimensional coordinates in said distorted image, from the distancefrom said origin in said distorted image, wherein, when saidthree-dimensional coordinates are represented by (x, y, z), and a radiusof said projection sphere is denoted by Rb, said composite indexcomputation unit determines said composite index Rn according to:Rn=√(x ² +y ²) if z≦√(x ² +y ²), andRn=√2×Rb−z if z<√(x ² +y ²).
 2. (canceled)
 3. The coordinate computationdevice as set forth in claim 1, wherein, when a zenith angle on saidprojection sphere is denoted by θ, and the distance from said origin ofsaid corresponding point in said distorted image is denoted by Rf, saiddistance computation unit determines the distance Rf from said origin,according to: $\begin{matrix}\lbrack {{Mathematical}\mspace{14mu} {Expression}\mspace{14mu} 12} \rbrack & \; \\{{Rf} = {{Rb} \cdot {F( {\cos^{- 1}\frac{\sqrt{{Rb}^{2} - {Rn}^{2}}}{Rb}} )}}} & \;\end{matrix}$ in a range in whichz≧√(x ² +y ²) is satisfied, and according to: $\begin{matrix}\lbrack {{Mathematical}\mspace{14mu} {Expression}\mspace{14mu} 13} \rbrack & \; \\{{Rf} = {{Rb} \cdot {F( {\cos^{- 1}\frac{{\sqrt{2} \times {Rb}} - {Rn}}{Rb}} )}}} & \;\end{matrix}$ in a range in whichz<√(x ² +y ²) is satisfied.
 4. The coordinate computation device as setforth in claim 1, wherein said distance computation unit has a lookuptable storing a relationship between an input value DN and an outputvalue DF which approximates, by a polyline, a relationship between saidcomposite index Rn and the distance Rf from said origin, finds, based onsaid composite index Rn computed by said composite index computationunit, DN[n], DN[n+1] satisfyingDN[n]≦Rn<DN[n+1], reads output values DF[n], DF[n+1] corresponding toDN[n], DN[n+1], and determines the distance Rf from said originaccording to: $\begin{matrix}\lbrack {{Mathematical}\mspace{14mu} {Expression}\mspace{14mu} 14} \rbrack & \; \\{{Rf} = {{{DF}\lbrack n\rbrack} + {\frac{( {{{DF}\lbrack {n + 1} \rbrack} - {{DF}\lbrack n\rbrack}} )}{( {{{Dn}\lbrack {n + 1} \rbrack} - {{DN}\lbrack n\rbrack}} )} \times ( {{Rn} - {{DN}\lbrack n\rbrack}} )}}} & \;\end{matrix}$
 5. The coordinate computation device as set forth in claim3, wherein said coordinate computation unit determines saidtwo-dimensional coordinates (p, q), according to:p=x×Rf/√(x ² +y ²)q=y×Rf/√(x ² +y ²) if Rf≠0; and according to:p=0q=0 if Rf≠0.
 6. An image processing device having: a coordinatecomputation device as set forth in claim 1; a spherical coordinatecomputation unit for taking each pixel in the output image as a pixel ofinterest, and computing the three-dimensional coordinates on saidprojection sphere corresponding to coordinates of the pixel of interest;a pixel value computation unit for computing a pixel value of said pixelof interest in said output image, based on the two-dimensionalcoordinates in said distorted image computed by said coordinatecomputation device; wherein said coordinate computation devicedetermines the two-dimensional coordinates representing a position insaid distorted image corresponding to the three-dimensional coordinatescomputed by said spherical coordinate computation unit, as thecoordinates representing a position corresponding to the coordinates ofsaid pixel of interest; and if there is a pixel at the position in saiddistorted image corresponding to the coordinates of said pixel ofinterest, said pixel value computation unit uses a pixel value of thepixel at said corresponding position, as the pixel value of said pixelof interest; and if there is no pixel at the position in said distortedimage corresponding to the coordinates of said pixel of interest, saidpixel value computation unit determines the pixel value of said pixel ofinterest by interpolation from a pixel value or pixel values of one ormore pixels in the neighborhood of said corresponding position.
 7. Acoordinate computation method that transforms three-dimensionalcoordinates on a projection sphere to two-dimensional coordinates on adistorted image, comprising: a composite index computation step forcomputing a composite index which is a combination of a height of apoint of interest on the projection sphere and a distance of the pointof interest from an optical axis which are obtained from saidthree-dimensional coordinates; a distance computation step for computinga distance from an origin in said distorted image, of a point in saiddistorted image corresponding to said point of interest, from saidcomposite index; and a coordinate computation step for computing thetwo-dimensional coordinates in said distorted image, from the distancefrom said origin in said distorted image, wherein, when saidthree-dimensional coordinates are represented by (x, v, z), and a radiusof said projection sphere is denoted by Rb, said composite indexcomputation step determines said composite index Rn according to:Rn=√(x ² +y ²) if z≦√(x ² +y ²), andRn=√2×Rb−z if z<√(x ² +y ²).
 8. An image processing method comprising:said composite index computation step, said distance computation step,and said coordinate computation step constituting the coordinatecomputation method as set forth in claim 7; a spherical coordinatecomputation step for taking each pixel in the output image as a pixel ofinterest, and computing the three-dimensional coordinates on saidprojection sphere corresponding to coordinates of the pixel of interest;a pixel value computation step for computing a pixel value of said pixelof interest in said output image, based on the two-dimensionalcoordinates in said distorted image computed in said coordinatecomputation step; wherein said coordinate computation step determinesthe two-dimensional coordinates representing a position in saiddistorted image corresponding to the three-dimensional coordinatescomputed in said spherical coordinate computation step, as thecoordinates representing a position corresponding to the coordinates ofsaid pixel of interest; and if there is a pixel at the position in saiddistorted image corresponding to the coordinates of said pixel ofinterest, said pixel value computation step uses a pixel value of thepixel at said corresponding position, as the pixel value of said pixelof interest; and if there is no pixel at the position in said distortedimage corresponding to the coordinates of said pixel of interest, saidpixel value computation step determines the pixel value of said pixel ofinterest by interpolation from a pixel value or pixel values of one ormore pixels in the neighborhood of said corresponding position.
 9. Thecoordinate computation device as set forth in claim 4, wherein saidcoordinate computation unit determines said two-dimensional coordinates(p, q), according to:p=x×Rf/√(x ² +y ²)q=y×Rf/√(x ² +y ²) if Rf≠0; and according to:p=0q=0 if Rf≠0.